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Abstract Body 

Limit 4 pages single spaced. 



Background / Context: 

Description of prior research and its intellectual context. 



Recent assessments indicate that American students do not score well on tests of scientific 
problem solving, relative to students in other nations. IMMEX is a web-based virtual 
environment that provides students with opportunities to solve science problems by viewing 
information resources through a suite of menu options, developing a hypothesis and submitting 
the hypothesis for feedback. Students’ patterns of search through the information sources are 
automatically classified by the IMMEX software using Hidden Markov Models, producing a 
quantitative measure of strategic efficiency. Prior work has found that many students do not use 
effective strategies in IMMEX and that they tend to retain poor strategies over time (Stevens, 
Beal & Sprang, 2009). That is, practice alone does not lead to better problem solving. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 



The goal of the study was to learn if students’ problem solving could be improved through the 
addition of metacognitive messages into the IMMEX simulation for chemistry. Prior work on 
the techniques used by effective human tutors indicates that they discuss how to approach a 
problem, how to monitor one’s progress and how to manage information resources effectively 
while solving a problem. The study question was whether messages based on a model of 
effective human tutoring could be integrated into the simulation and improve students’ strategic 
effectiveness. 

Setting: 

Description of the research location. 



The study was conducted in chemistry classes in a large Southern California school district in 
which IMMEX is part of the science curriculum. The district serves a diverse student population 
(35% Hispanic, 48% White, 15% Asian American, 2% African American). Science teachers 
receive professional development training in integrating IMMEX into their classes (Thadani, 
Stevens & Tao, 2009). Teachers set up student accounts in IMMEX and are free to use it 
whenever they want to during the academic year. Most teachers schedule 6-8 sessions over the 
course of the academic year. 



Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

The study included Grade 9 students in chemistry classes who participated as part of their 
classroom work. The sample used in the analyses included students who completed at least five 
cases in the Duck Run problem set. There were 195 students who used the message-enhanced 
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version of IMMEX, and 173 students who used the original version. (Students were randomly 
assigned by the server at the time of log in to one of the two versions.) 



Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

For Track 2, this may include the development and validation of a measurement instrument. 

Students worked either with a version of IMMEX that included metacognitive messages 
delivered at the start of each case, or with the original version that did not include the messages. 
In the message-enhanced version, a brief text message was shown on the screen at the start of 
each problem in the set. Messages were delivered randomly across students, with the restriction 
that each message would be shown only once to an individual. 

Examples of messages included: "When you work on a hard problem, sometimes you're going to 
feel confused. That's a sign your brain is trying out different ideas. Don't give up too soon on 
this case." AND "If you get confused on this case, ask yourself, 'What do I need to know to make 
progress, and where can I find that information?' It's fine to ask for help, but make sure you 
know exactly what you need help with." The messages focused on strategies for problem 
solving rather than on specific chemistry content. 

In the study, students worked with the “Duck Run” problem set in which the objective is to use 
available information to identify an unknown substance for proper removal. Information sources 
include state of matter (solid, liquid, gas), reports from observers about characteristics, results of 
chemical tests performed on the substance, number of electrons, etc. Students can view the 
information sources in any order and have up to two tries to answer correctly. The Duck Run 
problem set includes 12 different versions (cases) with different unknowns and associated 
information sources. Cases vary in difficulty and are delivered in random order. Each case takes 
approximately 15 to solve on average. 



Research Design: 

A between-subjects design was used: Students used either the original version of IMMEX as part 
of their chemistry coursework, or a version that included metacognitive messages. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

As students solved cases in IMMEX, their actions, including selection of information sources to 
view, the order in which different sources of information were viewed, and the time taken on the 
case were automatically recorded, along with the problem outcome (correct on the first or second 
attempt, or not correct). A Hidden Markov Model is used to classify the student’s strategic 
efficiency on each case. The Strategic Efficiency Index is a quantitative measure of the extent to 
which the student views the appropriate information sources, does not return to view information 
that has already been seen, and whether the correct answer is chosen in one or two attempts (or 
not at all). The SEI metric is automatically computed by the IMMEX software. Data for each 
student included the SEI score on each case, as well as answer accuracy. 
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Findings / Results: 

Description of the main findings with specific details. 



Results indicated that there were no differences in initial performance in IMMEX, meaning that 
the two groups scored similarly on the first case. However, students who worked with the 
message-enhanced version showed more improvement across cases, meaning that they became 
more efficient in their problem solving, as well as more likely to find the correct solution. The 
differences between the groups were most apparent on the most difficult cases (difficulty was 
established in prior research with IMMEX). Additional analyses focused on the performance of 
students in both groups who continued to use inefficient problem solving strategies, such as 
looking through information sources haphazardly, consulting information not relevant to the 
current case, and looking at the same information multiple times. On the fourth and fifth cases, 
there were relatively few students who continued to use poor strategies but those who were in the 
message-enhanced IMMEX group were more likely to solve the cases correctly. This suggested 
the possibility that the messages promoted better integration of the information even among 
students with the weakest skills. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 



The findings suggest that students can be assisted to adopt better problem solving strategies 
through relatively simple changes to a technology-based learning environment. The addition of 
messages designed to encourage students to think about their actions, set goals and monitor 
progress, engage in appropriate help-seeking and tolerate a certain amount of frustration during 
problem solving appeared to be beneficial, especially for students who were not doing 
particularly well in the activity. 



SREE Fall 2011 Conference Abstract Template 



B-6 




Appendices 

Not included in page count. 
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